Papers with multimodal encoder
Grafting Pre-trained Models for Multimodal Headline Generation (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Existing approaches to generate video headlines with pre-trained language models are labor intensive and impractical. |
| Approach: | They propose to graft the encoder from the pre-trained video-language model on the generative pre-trainer model and propose a consensus fusion mechanism for the integration of different components. |
| Outcome: | The proposed model achieves strong results on a brand-new dataset collected from real-world applications. |
Fine-grained Image Captioning with CLIP Reward (2022.findings-naacl)
Copied to clipboard
| Challenge: | Modern image captioning models are usually trained with text similarity objectives . reference captions often describe only the most salient objects in images . |
| Approach: | They propose to use CLIP to calculate multi-modal similarity and use it as a reward function . they propose a simple finetuning strategy to improve grammar that does not require extra text annotation. |
| Outcome: | The proposed model generates more distinctive captions than the CIDEroptimized model on text-to-image retrieval and fineCapEval. |
Improving Grammatical Error Correction with Multimodal Feature Integration (2023.findings-acl)
Copied to clipboard
| Challenge: | Experimental results show that multimodal GEC models improve over strong baselines and achieve a new state-of-the-art result on the Falko-MERLIN test set. |
| Approach: | They propose a framework that integrates both speech and text features to enhance GEC by generating audio from text using advanced text-to-speech models. |
| Outcome: | The proposed framework improves on CoNLL14, BEA19 English, and Falko-MERLIN German datasets. |
Combo of Thinking and Observing for Outside-Knowledge VQA (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to visual question answering use external knowledge to acquire and use knowledge beyond images. |
| Approach: | They propose to constrain the cross-modality space into the same space of natural-language space . they propose a multimodal encoder, textual encoder and answer decoder to introduce more types of knowledge . |
| Outcome: | The proposed framework outperforms the state-of-the-art by 6.17% accuracy on a cross-modal space and natural-language space. |
Retrieval over Classification: Integrating Relation Semantics for Multimodal Relation Extraction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to multimodal relation extraction ignore structural constraints and lack semantic expressiveness for fine-grained relation understanding. |
| Approach: | They propose a framework that reformulates multimodal relation extraction as a retrieval task driven by relation semantics. |
| Outcome: | The proposed framework achieves state-of-the-art performance on the benchmark datasets MNRE and MORE and exhibits stronger robustness and interpretability. |
Learning Dynamic Representations and Policies from Multimodal Clinical Time-Series with Informative Missingness (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to accommodate missingness in clinical time series, but how to extract and use information carried by the observation process itself remains underexplored. |
| Approach: | They propose a patient representation learning framework that leverages informative missingness to learn multimodal clinical time series from structured and textual data. |
| Outcome: | The proposed framework improves offline treatment policy learning and adverse outcome prediction on ICU sepsis cohorts from MIMIC-III, MIMIC IV, and eICU. |